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Abstract 

We discuss a general notion of "sparsity structure" and associated recoveries of a sparse signal 
from its linear image of reduced dimension possibly corrupted with noise. Our approach allows 
for unified treatment of (a) the "usual sparsity" and "usual ii recovery," (b) block-sparsity 
with possibly overlapping blocks and associated block-^i recovery, and (c) low-rank-oriented 
recovery by nuclear norm minimization. The proposed recovery routines are natural extensions 
of the usual ii minimization used in Compressed Sensing. Specifically, we present nullspace-type 
sufficient conditions for the recovery to be precise on sparse signals in the noiseless case. Then 
we derive error bounds for imperfect (nearly sparse signal, presence of observation noise, etc.) 
recovery under these conditions. In all of these cases, we present efficiently verifiable sufficient 
conditions for the validity of the associated nullspace properties. 

1 Introduction 

We address the problem of recovering a representation w = Bx G -E of an unknown signal x & X via 
noisy observations 

y = Ax + 

of X. Here X, E are Euclidean spaces, A : X and B : X ^ E are given linear sensing and 

representation maps, and ^ is "uncertain-but-bounded" observation error satisfying (f){C,) < e (</)(■) is 
a given norm on R"^, and e is a given error bound). We consider, for instance, the recovering routine 
of the form 

y H- x{y) G Argmin : 0(Au — y) < e} w{y) = Bx{y), 

and we want this recovery to behave well provided that Bx is sparse in some prescribed sense. In this 
note, we introduce a rather general notion of sparsity structure on the representation space E which, 
under some structural restriction on the norm || ■ ||, allows to point out "nullspace type" conditions 
for the recovery to be precise provided that Bx is "s-sparse" with respect to our structure. It also 
allows for explicit error bounds for "imperfect recovery" (noisy observations, near s-sparsity instead 
of the exact one, etc.) The motivation behind this construction is to present a simple unified general 
framework, which allows, for instance, for a simple treatment of three important particular cases: 
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• recovering s-sparse in the usual sense (at most s nonzero entries) signals via ii minimization 
(the corresponding nullspace property goes back to [4, 13]) 

• recovering s-block-sparse signals via block-£i minimization, and 

• recovering matrices of low rank via nuclear norm minimization. 

We present the respective sparsity structures and provide verifiable sufficient conditions for the 
validity of associated nullspace properties (and thus - for the validity of the corresponding recovery 
routines); the prototypes of our verifiable conditions can be found in [5, 6, 7, 8]. 

2 Problem description and recovery routines 
2.1 Situation 

Let X, E he Euclidean spaces, x i— >■ Ax : X — )■ R"^ be a linear sensing map, and x ^ Bx : X ^ E 
be a linear representation map. We are interested in the problem as follows: 

(!) Given a noisy observation 

y = Ax + ^ (1) 
of an unknown signal x G X, we want to recover the representation w = Bx of x. 

Sparsity structure We will focus on the case when a priori information on x is expressed in terms 
of properly defined sparsity of the representation Bx of x. To this end, we define a sparsity structure 
on E, specifically, as follows: 

Let II "11 be a norm on E, || ■ ||* be the conjugate norm, and V be a family of linear maps 
of E into itself, such that 

A.l. Every P is a projector: = P. 

A. 2. Every P eV is assigned a nonnegative weight v{P) and a linear map P on E such 
that PP = 0; 

A. 3. Whenever P eV and f,g G E, one has 

||P7 + ^*^7||*<max[||/||„||(?||,], (2) 

where for a linear map Q acting from a Euclidean space E into a Euclidean space 
F, Q* is the conjugate mapping acting from F to E. 

A collection of the just introduced entities satisfying the requirements A. 1-3 will be 
referred to as a sparsity structure on E. 

From now on, given a sparsity structure, we set for a nonnegative real s 

Vs = {PeV: v{P) < s}. 

Given s > 0, we call a vector w E E s-sparse, if there exists P eV such that i'{P) < s and Pw = w. 
We call a vector x E X a.s s-sparse, if so is its representation w = Bx. 

We are about to present some instructive examples of the just outlined situation. Given a finite 
set V, we denote by R(^) the space of real vectors with entries indexed by elements of V and equip 
this space with the standard inner product. For a subset S' G we set R(5') = {v E R(^) : Vi = 
OVi ^ S}, and refer to R(5') as a coordinate subspace of R(^). By Ps, S C V, we denote the 
coordinate projector - the natural projector of R(l^) onto R(5'). 
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2.2 Examples 

Example I. a: ii recovery In this example, 

• X = E = R" = Ti{V := {1, n}) with the standard inner product; 

• The representation map is the identity: Bx = x; 

• V is comprised of projectors onto all coordinate subspaces of R", P = In — P', 

• z/(P) = Rank(P); 

Properties A. 1-3 clearly take place, s-sparsity of x G X as defined above is the usual sparsity (at 
most s nonzero entries), and (3) is the usual £i recovery. 

Example I.b: Group ii recovery Now we want to model the "block-sparse" situation as follows. 
X = R" = R(y = {1, 77,}), and the index set V is represented as the union of K nonempty 
(and possibly overlapping) subsets Vi,...,Vk, so that to every x E X one can associate blocks x^, 
i = 1, ...,K, which are natural projections of x onto Ti{Ve). Assuming the subsets to be assigned 
with positive weights xe^ we define the sparsity of a signal x G A as the "weighted number of 
nonzero blocks x^," that is, the quantity s{x) = ^i-^ij^oXe- This can be modeled by the following 
representation and sparsity structure: 

• E = R(Vi) X ... X K{Vk), so that w e E is a. block vector [w^; ...-,10^] with G R(V^), and 

Bx = [x^ G E; 

• V is comprised of orthoprojectors P = Pj onto the subspaces Ej = {[w^; w^] E E : = 
OWi ^ /}, associated with subsets / of the index set {1,..., A}, and P = Id — P, where Id 
stands for the identity mapping on A; 

• '^(Pi) = Efe/ 

• The norm || ■ || is defined as follows. For every i < K, we equip R(V£) with a norm || ■ and 
set 

\\[z\..,z^]\\ = f2\\z%e, 

1=1 

Verification of A.1-A.3 is immediate. Note that when Vi,...,Vk do not overlap and Xi = ^ fo^' ^.H 
i, we find ourselves in the standard block-sparse situation: E can be naturally identified with A, 
making B the identity, vectors from A = R" are split into K non-overlapping blocks, the norm is 
the block-£i norm, and s-sparsity of x means that x has at most s nonzero blocks. 

Example II: Nuclear norm recovery In this example, 

• X = E = IV^'^ with p > q and the Frobenius inner product, B is the identity; 

• V is comprised of the mappings P(x) = PieftXPnght, where Piea G R^^p and P^ght G R'^^'^ are 
orthoprojectors, and P(x) = (Jp - Pi,tt)x{Ig - P,ight);^ 

"'^To avoid notational ambiguity, in the situation of Example II we denote the image z E E under a mapping 
P (z V hy P{z) rather than by Pz, to avoid coUision with the notation for matrix products hkc (Ip — Picft)xilq — -Pright)- 



3 



• II ■ II is the nuclear norm: ||a;|| = X]j=i'^i(^)' where (Ti{x) > a2{x)... > crg(x) are the singular 
values of X e R^^'^. 

The assumptions A. 1-3 clearly take place. 

To verify A. 3, observe that the norm conjugate to || • || is the spectral norm ||x||2,2 := o'i(a^)- 
We now have 

||/||2,2<1, ||ff||2,2<l 

ll^*(/)l|2,2 = ||Pleft/^HgM||2,2 < 1, \\P\9)h,2 = Wilp - - Pr:,^^)h,2 < 1 

\\P*{f) + P*{g)h,2<h 

where the last follows from the fact that the orthogonal complements to the kernels of P*{f) 
and P*{g), same as the images of these matrices, are orthogonal to each other. 

In this case, by assigning the projectors P eV with weights according to 

u{P) = max[Rank(Pi,ft),Rank(PHght)], 

we arrive at the situation where s-sparse signals are exactly the p x q matrices of rank < s. 

3 Main results 

3.1 Recovery routines 

Let be a norm on the image space of A, and let e be an a priori upper bound on the 0-norm 0(^) 
of the observation error, see (1). In order to recover the representation Bx of signal x underlying the 
observation (1), we use regular recovery - the "standard" recovery by || ■ || -minimization as follows: 

y ^ Xrcgiy) e Argmin {||5m|| : (j){Au - y) < e} ^-^ w.^giy) = Bx,cg{y) (3) 
uex 

and we treat w^cgiu) as the resulting estimate of Bx. 

We say that the sensing map A is s-good if the above recovery in the noiseless case 
(e = 0) reproduces exactly the representation Bx of every s-sparse signal x. 

We also consider an alternative to (3), specifically, the penalized \\ ■ \\ recovery introduced in [7, 8]. 
This recovery routine is given by 

y ^ ^pcn(y) e Argmin{||fin|| + \(j){Au - y)} ^ Wpcn{y) = Bxp^niy) (4) 
where A > is a penalty parameter. 

3.2 s-goodness and nullspace property 

We start with the following immediate observation: 
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Lemma 3.1 In the situation of section 2.1, let w ^ E, P ^ V be such that Pw = w. Then for every 
z & X one has 

\\w + Bz\\ > \\w\\ + \\PBz\\ - \\PBz\\. (5) 
In particular, if the following "nullspace property" holds true: 

VP eVs, ze Ker(A), Bz^Q: \\PBz\\ > \\PBz\\, (6) 

then A is s-good. 

Proof. Let w & E and P eV he such that Pw = w. We have 

\/{f,g : max{||/||„ \\g4 < 1) : ||P7 + < 1 [by A.3] 

which imphes 

=^\\w + Bz\\ > ||w + P^IIIlP*/ + ^*fi'll* 

> (P7 + P*g, w + Bz) = (/, Pw + PBz) + {g, Pw + PBz) 

> (/, w) + (/, PBz) + {g, PBz) [since w = Pw and therefore Pw = PPw = 0] 

> {f,w)-\\PBz\\ + {g,PBz). 

When choosing f,g to be such that ||/||* = \\g\\* = 1 and (/, w) = {g,PBz) = ||PP2;||, we get 

\\w + Bz\\ > \\w\\ - \\PBz\\ + IIPP-^II, 

as claimed in (5). 

Now let (6) take place, and let us prove that A is s-good. All we need to verify is that if x is 
s-sparse and x = x{Ax), see (3), then Bx = Bx. Setting z = x — x, so that z G Ker(74), choosing 
P EV such that z/(P) < s and PBx = Bx and applying (5) with w = Bx, we get 

||Px|| = \\Bx + Bz\\ > \\Bx\\ + IIPP^II - IIPP^II, 

while by the origin of x we have \\Bx\\ < ||Px||. It follows that ||PP2;|| < ||PP2;||, which, by the 
nullspace property (6), is possible only when Bz = (recall that z/(P) < s and z G Ker(74)). | 

Remark 3.1 Independently of any assumptions on || ■ ||, an evident necessary condition for s- 
goodness of A is: 

Whenever P E V, ^{P) < s, x E X are such that PBx = Bx, and z G Ker(/1), Bz ^ 0, 
there exists f, ||/||* = 1, such that {f,PBx) = ||PPx|| and {f,Bz) > 0, 

When modifying this condition by replacing (/, Bz) > in the conclusion with (/, Bz) > 0, this 
necessary condition for s-goodness becomes sufficient. This, by the way, immediately implies the 
necessity and sufficiency of the standard nullspace property [4, 13] for the validity of ii minimization 
and the translation of it to the matrix case given by [9, 10]. Moreover, recently, [2] has established 
the sufficiency of this condition in the case of decomposable norms, i.e., sparse, non-overlapping 
block-sparse and low-rank recovery following a unified view based on subdifferentials. 
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3.3 Error bounds for imperfect || • || recovery 

Conditions 0^(7, and C^(7,/3;0). In the sequel, we shall use the following two conditions 
(where s, 7 > 0, /3 G [0, 00], and 0(-) is a norm on the image space of A): 

C.(7,/3;0): 

y{z eX,P eVs): \\PBz\\ + \\Bz\\ - \\PBz\\ < f3(f){Az) + -f\\Bz\\ 

(from now on, (+00) -0 = and (+C)o) ■ a = +00 when a > 0). 

C^(7,/3;0): there exists a (semi)norm || ■ on E such that 



(a) V(2 eX,PeVs): \\PBz\\ + \\Bz\\ - \\PBz\\ < \\Bz\\j,^ 

(b) \fzeX : \\Bz\\v, < (3(j){Az) + l\\Bz\\. 



(7) 



Let us make the following immediate observations: 

Remark 3.2 (i) The validity of condition 0^(7, with some 7 < 1, and some /3, implies the 
validity of the nullspace property (6); 

(ii) C+ (7, /3; 0) implies C, (7, /3; 0) ; _ 

(iii) The (semi)norm := max [||Pw|| + || (Id — P)tf;||] satisfies (7. a) (since by Triangle 

inequality we have \\PBz\\ + \\Bz\\ - \\PBz\\ < \\PBz\\ + \\{ld -P)Bz\\); 
{w) Whenever s' < s, 7' > 7, and > /3 one has Cj"(7, /3; 0) =^ 0^(7', /?'; 0), and similarly for 
the condition Cg/ 

(iv) Condition C'^ {'-f , +00] (f)) reads 

There exists a (semi)norm || • ||-p^ on E such that 

(a) V(z eX,P eVs): \\PBz\\ + - \\PBz\\ < 

(6) VzGKer(A): \\Bz\\v^ < -i\\Bz\\. 

if this condition is satisfied, then for every 7' > 7 there exists (3 < 00 such that the condition 
0^(7', /3;0) is satisfied. 

Error bounds for imperfect regular || ■ || recovery are stated in the following 

Proposition 3.1 In the situation of section 2.1, let a sparsity level s > be given, and let the 
condition 0^(7, /3;0) take place for some 7 < 1 and (3 < 00. Given a signal x G X along with 
its observation y = Ax + ^, where 0(^) < e, let x be "nearly s-sparse" and x be "nearly Xrcg{y)," 
specifically, for given nonnegative tolerances 6x, S^, and 6 one has 

• there exists P EVg such that ||(/ — P)Bx\\ < 5x ("near s-sparsity of x"); 

• one has 

(a) <j){Ax - [Ax + ^]) < e + 6^ 

(6) \\Bx\\ <0pt + 5, Opt:=mm{\\Bu\\:(j){Au-y)<e}. V»j 

u 

("x is a nearly feasible nearly optimal solution to the optimization problem specifying x{y) "). 

Then 

Proofs of the results of this section are put in the appendix. 
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Error bounds for penalized || ■ || recovery 

Proposition 3.2 In the situation of section 2.1, let s >0, '-f < 1, </>(■) and (5 < oo be such that the 
condition 0^(7, is satisfied. Let also the penalty parameter A in (4) be > (5. Finally, let the 
signal x underlying the observations be "nearly s-sparse, " meaning that there exists P EVg such that 
II (J — P)Bx\\ < 6ri;, and let x be a near-optimal solution to the optimization problem specifying Xpcn, 
namely, 

\(j){Ax-y) + \\Bx\\ < min{A0(A2 - y) + \\Bz\\} + 6. 

Z 

ThcTi 

2i^+* + 2A^ (10) 

where ^ = y — Ax is the observation noise. 

Comment: As compared to the plain || ■ ||-recovery (3), the penalized || ■ || recovery requires a priori 
knowledge of a /3 < 00 such that the condition 0^(7,/?; </>) takes place; indeed, in order for the error 
bound (10) to be applicable, we should ensure A > /3. As a compensation, the penalized || ■ || recovery 
does not require any a priori information on the level of observation error, and as such is well suited 
for the case when the latter is random (or a sum of a random and a bounded components). 



4 Application examples 

In the rest of this note, we are interested in the particular forms taken by the nullspace sufficient con- 
dition for s-goodness of A (Lemma 3.1) and the error bound for imperfect || • ||-recovery (Proposition 
3.1) in the examples described in section 2.1. 

4.1 Example I. a: £1 recovery 

In the situation of Example La, the nullspace property (6) reads 

7,(A) := max{||x||,,i : x G Ker(v4), ||x||i < 1} < 1/2, (11) 

X 

where ||a;||s,i is the sum of the s largest magnitudes of entries in x. This is a well-known necessary 
and sufficient condition for the validity of the standard sparse ii recovery [4, 13]. It is immediately 
seen that condition C^(7, /3; (p) is satisfied if and only if it is satisfied when setting \\z\\-p^ = 2\\z\\s,i, 
and in this case the condition is equivalent to 0^(7, /3; 0). The latter condition reads 

V(2GR"): \\z\U,<^<PiAz) + ^\\z\U. (12) 

Validity of this relation is equivalent to the fact that the quantity 7s (A, ■) introduced in [5] satisfies 
7s(A, /3/2) < 7/2 (see [5, Theorem 2.2]. What is denoted by || ■ || in the latter reference, is now 0(-)), 
and with this in mind, error bounds (9) and (10) recover the bounds in [5, Theorem 3.1]. Beside 
this, one can find in [5] verifiable sufficient conditions for the validity of 0^(7, /3;0), their relations 
to Restricted Isometry Property, etc. 
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4.2 Example I.b: Group £i recovery 

In the situation of Example I.b, given a positive real s, let us define the norm Hs{-) on as 

{0,1} V£, Yl 

and let || ■ be the norm on E given by 

\\w\\i,s = 7rs([||wl(i); 11^^11(2); \\w^\\{k)])- 
Observe that for every z E X and every P = Pi E Vs v^e have 

K 

\\PBz\\ + 115.11 - \\PBz\\ = \\iBz)%,^ + J2 \\iBz)%e) - 

eel e=i i^i 

= 2Y\\iBz)%,) < \\Bz\U,,, 
eel 

where the concluding < is given by Pj G Vs, so that J^eei^f^ — ^- Thus, with || • \\-p^ set to || ■ Hi^^ 
the condition (7) is satisfied. We have arrived at the following 

Proposition 4.1 In the situation of Example I.b, for every positive reals s, /3,7, the condition 

MzeX : \\Bz\\i^s < (3(j){Az) + -f\\Bz\\ (13) 

is sufficient for the validity o/ C^(7, /3; 0), and thus — for the validity o/ Cs(7, /9; 0). As a result, 
condition (13) with j < 1 is sufficient for s-goodness of A and for the applicability of the error bounds 
(9) and (10). 

A verifiable sufficient condition for (13) Condition (13) is difficult to verify. We are about 
to present a verifiable sufficient condition for the validity of (13) inspired by [8]. For a linear map 
Q'^^ : R(V£) —7- R(Vfc), let ||Q'^^||{£fc) be the norm of the map induced by the norms || ■ ||(^) and || ■ \\{k) 
on the argument and the image spaces: 

\\Q^^\\[tk) = max{||Q'^^n||(fc) : ||n||(^) < 1}. 

u 

Let also be the dimension of E, and let ni, nx be the cardinalities of Vi, Vk, so that E can 
be identified with R^ = R"i x ... x R"*'. For an x matrix W, let W'^^ be the x ng blocks 
of W associated with the direct product representation R"^ x ... x R"^ of R^, and let Q[W] be the 
K X K matrix with the entries fi^^ = 1 < k,i < K. 

Proposition 4.2 In the situation of Example I.b, given 7 > 0, let m x N matrix H and N x N 
matrix W satisfy the relations 

(a) B = WB + H^A 

(b) max,<^7r,(Col,(fi[iy])) <7, ^ ^ 
where Colj{Q) is the j-th column of matrix Q. Then the relation (13) holds true with 

13 = I3[H] := max : 4>{v) < l] . 



8 



Proof. Let z G X. Under the premise of the proposition, we have 

K 



e=i 

K 



i=i 



\Bz\\i^,, < \\Bz\\ma.x7rs{Coh{n[W])) + \\H^ Az\\i^ 



and the desired conclusion follows. 



Discussion The sufficient condition for the validity of (13) stated in Proposition 4.2 reduces to 
solving a system of convex constrains in matrix variables H, W and scalar s 7, /3, namely, 

B = WB + H^A, vr, {Coh{yt[W])) < 7 V£, := max \\H^v\\i^s < (15) 

v:<j>{v)<l 

This system, although convex, still can be difficult to process, since the convex functions ||W^'^^||(£fc), 
i^s{') and ^ s{H) can be difficult to compute. In such a case, one can replace these functions with 
their efficiently computable upper bounds (for details, "solvable cases," etc., in the case of = 1 
see [8]). For instance, (15) is computationally tractable when 

• all norms || ■ are either £1, or or £2 norms (this makes the matrix efficiently 
computable); 

• in appropriate scale, all weights ^'^^ integers from a once for ever fixed (or polynomially 
growing with problem's sizes) range, which makes the norm vr^ efficiently computable. Note 
that one can always replace tt^ with a reasonably tight upper bound on vr^, specifically, the 
norm 



K K 

2max< Y^'q(\ue\ : < rje < min[l, Floor(s/x£)] V^, '^XeVi < ^ > ; 



• 0(-) is the ii norm. 

The last assumption is indeed restrictive. It, however, is responsible solely for the tractability of the 
constraint ^'s(-ff) := max < /3, and is crucial only if one's objective is to compute an 

v:<f>{v)<l 

upper bound on /3. On the other hand, it is of primary importance to ensure 7 < 1, since otherwise 
Proposition 3.1 provides no error bound at all. 

Finally, we refer the reader to [8] for details on relationship of the derived conditions with block- 
RIP and other sufficient conditions used in the Compressive Sensing literature. 
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4.3 Example II: Nuclear norm recovery 



For a p X q matrix z with p > q, let 



k 



^k{z) = ^ai{z), 1 < k <q. 



i=l 



Observe that in the situation of Example II, where B is the identity, and the sparsity parameter s 
can be w.l.o.g. restricted to be a nonnegative integer, we have (everywhere in this section, || ■ || is the 
nuclear norm). 



Indeed, let P G Vs, so that Rank(Pioft) < s and Rank(P,ight) < s. Then ||P(2;)|| = ||-Pieft-2-Pright|| < 
T,s{z) by the Singular Value Interlacement Theorem. Since the matrix P{z) differs from z 
by matrix of rank at most 2s, by the same Singular Values Interlacement Theorem we have 
(^i{P{z)) > ai+2siz), whence ||P(z)|| > ||2;|| - J^2siz). 

We have arrived at the following 

Proposition 4.3 In the situation of Example II, the norm 

WzW-p^ := J:s{z) + E2s{z) 
on X = E = W^'^ satisfies the condition {7. a), so that the condition 



is sufficient for the validity o/ C^(7, /3; 0), and thus — for the validity o/ 0^(7, /3; 0). As a result, 
condition (16) with 7 < 1 is sufficient for s-goodness of A and for applicability of the error hounds 
(9) and (10). 

Clearly there is a gap between the above sufficient condition and the necessary nullspace condition 
for exact low-rank matrix recovery, which is 

2S,(2) < II 2 II for aU z G Ker(A), 2^0. 

On the other hand our sufficient condition is stronger than, the only known to us, sufficient condition 
given in [10], which requires 

2T.2s{z) < \\z\\ for aU z E Ker(A), z^O. 

A verifiable sufficient condition for (16) Following the same exposition scheme as in Examples 
I.a-b, it is now time to point out a verifiable sufficient condition for the validity of (16). Let if be a 
linear map from X = R^^"? into R™, so that W = Id — H*A, Id being the identity mapping on X, 
is a linear map from X into X. Assume that W satisfies the requirement 



V(2eRP^^PGn) : ||P(2)|| <S,(2), \\P{z)\\>\\z\\-J:2s{z). 



yz G R^''^ : 



S,(z) + S2.(z) </30(Az)+7||z 



(16) 



\fzeX : 



j:s{Wz) + j:2s{Wz)<j\\z\\- 



(17) 



we claim that then (16) holds true with 



/3 = /3[H] = max {S,(iJ*t;) + J:2s{H*v) : 0(t;) < 1} . 



(18) 
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Indeed, let z £ X. Assuming (17), and setting ^{-01) = + T,2siw), w E RP^"?, we have 

7r(z) = tt{Wz + H*Az) < tt{Wz) + 7r{H*Az) < 7||z|| + /3[i/](^(Az) 
as required in (16). 

The question is, how to efficiently certify the vahdity of (17). The proposed answer is as follows. 
Note that 

Sfc(u^) = max{TT{wh^) : \\h\\ < k, \\h\\2,2 < 1}, 

h 

therefore (17) is exactly 

7 > Opt[iy] := max {S,(iyz) + S2,(PFz) : \\z\\ < 1} 

= max {TT{[Wz][u + vf): \\z\\<l, Si ^ "^"f -^'^ V 

Now, 

pq 

where 0[iy] is a properly defined linear in W pq x pq matrix, and (8> is the Kronecker product; in 
other words h'^ ^ z is the pq x pq matrix with p x q blocks [h'^ ® z]^'^ = h^^z, l</i<q', l<z^<p. 
We conclude that 



pq 

Opt[W] < max <j J2 + ^^j] : W G Z„ V G Z2. 



where 



/i G Hfc = {/i G Rf^^ : \\h\\ < k, \\h\\2,2 < 1} 
U7 G W = {w G Rf''^ : < 1} 

When replacing with a computationally tractable set Z^ D Z^ and setting 



pq 



we obtain an efficiently computable convex in W upper bound on Opt[iy], so that the efficiently 
computable convex constraint 

Opi[W] < 7 (20) 

implies the validity of (17). We are about to point out two computationally tractable convex relax- 
ations Zl D Zfc of the sets Z^. 

Our first observation is as follows: since the singular values oi h?' ® z are pairwise products of 
singular values of h and 2, the set 

Ufc = {W G R^'^^^P" : \\U\\<k, ||W||2,2<1} 

contains Z^, so that we can set Z^ = Ufc. As a result, the efficiently computable convex constraint 



Opt[Id-ff*A] < 7 (21) 
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on the matrix variable where Opt is given by (19) with = Ufe, is a verifiable sufficient condition 
for the validity of (17). 

Unfortunately, the sufficient condition in (21) is really poor. Note that in the context of Com- 
pressive Sensing, the only interesting case is the one of 7 < 1. We are about to show that (21) can 
certify the validity of (17) with 7 < 1 only in extreme cases: 

Proposition 4.4 In the situation in question, it holds 



In other words, when the dimension of Ker(y4) is of order of the dimension pq of X , our verifiable 
sufficient condition for the validity of the nuclear norm recovery can certify s-goodness of A only for 



The proof of the proposition is provided in the appendix. 

In fact, the tractable convex relaxations i— Ufc underlying the condition (21) can be improved, 
resulting in a weakened form of (21). Specifically, let M'[h,w] = h ® w & RP^" . Obviously, if 
M[h,w] = ®w, then M'[h,w] = M'{M[h,w]) for certain linear map M' : Rp^^p" -> Rp'x^' 
induced by an appropriate x q^ rearrangement of the entries in a x pq matrix. Using again the 
fact that the singular values of /i ® w are the pairwise products of the singular values of h and w we 
conclude that the set 



contains Z^. Further, let f{h) = Col{h^) = [Rowi(/;,)"^; KoWp^h)'^], where RoWj(/i) is the i-th row 
of h] let also g{w) = Col(w) = [Coli{w), ...,Colg{w)], where Co\j{w) is the j-th column of w. Let 
us denote M"[h,w] = f{h)g{w)'^. It is immediately seen that M"[h,w] = J^"{M[h,w]) for certain 
linear mapping Ai" : IV^^p'^ — )■ RP^xp? induced by an appropriate permutation of the entries in a 
pq X pq matrix. Furthermore, 




(22) 



. = 0(1). 



Vfe = {W G Ufc : ||A^'(W)|| < k, \\M\m2,2 < 1} 



max<^ \\M"[h,w]\\ 



heHk = {he RP""^ : \\h\\ < k, \\h\\2,2 < 1} 

w eW = {w eRP""^ : \\w\\ < 1} 



< Vk. 




M"[h,w]=f{h)g{wf= Y^fih') 



where f{h^) = [a{b^; ...\a^p}/-\ and g{vS) = [abi; ...;a6q]. We see that f{h^) are mutually orthogo- 
nal unit pg-dimensional vectors, and g{w) is a unit p(7-dimensional vector, so that M" is a rank 
1 matrix with the maximal singular value Vk. Thus, ||M"[/i, w]|| < y/k, as claimed. | 
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Let now 



Z* = {WGVfe: \\M"m\<Vk} 



< 1, \m2,2 < 1; 
||A^'(W)|| < A;, ||A^'(W)||2,2<1; 
\\M"{m<Vk 



We conclude from the above that Z^ is a convex set which cointains Z^ and is contained in U^. Thus 
the function 



pq 

Opt*[W] = max <j J2{Q[W]),,[Uij + V,,] : UeZ^Ve Z^ 
"'^ [i,j=i 

is an efficiently computable upper bound on Opt[Vr] which is < We arrive at 

Proposition 4.5 If I3[H] is an efficiently computable convex in H upper bound on /3[H], then solv- 
ability of the system of efficiently computable convex constraints 

Opt* [Id - H*A] < 7, ^[H] < P 

in matrix variable H , f3 and 7 being parameters, is a verifiable sufficient condition for the validity of 
(16). 



Discussion Though we do not know at this point whether the verifiable condition of Proposition 4.5 
for the validity of (16) still obeys the "limits of performance" of Proposition4.4, numerical experiments 
show that the computationally tractable set Z^ is a proper subset of U^. Moreover, no one of the 
constraints in the description of Z^ is redundant, meaning that dropping any one of them can strictly 
increase the solution set. At the moment is the smallest computationally tractable relaxation of 
Zfc known to us. 

Note that the verifiable sufficient conditions we have presented for Examples I.a-b can certify 
s-goodness of a sensing map for s as large as 0(-y/m) (for details, see [5, 8]). On the other hand, 
it should be noted that not so good (or even really bad, as in the case of verifiable condition (21)) 
"limits of performance" of our verifiable sufficient conditions for the validity of sparsity/low rank 
oriented recovery do not automatically mean that these conditions are of small or no interest. The 
point is that the recovery in question makes perfect sense also in the case when the observations are 
not "deficient", that is, the sensing map A has the trivial kernel {0}. Since such an A can be poorly 
conditioned, the question of how well can we recover (representations of) sparse/low rank signals 
remains nontrivial and important even in this "fully observable" case. Here our verifiable sufficient 
conditions, at least in the case of B = Id, can certify the validity of, say, C^(l/2, (3, 0) for all s and 
all (3 > (3{s), with some efficiently computable /3(s), and thus we can efficiently upper-bound the 
recovery errors as functions of signal's sparsity. 
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A Proofs 

A.l Proof of Proposition 3.1 

Let P G be such that || (/ — P)Bx\\ < 6^, and let w = PBx, w = Bx — PBx, z = x — x. We have 

(a) Pw = P^Bx = PBx = w [since all P G "P are projectors] 

(6) ||Px|| < Opt + 5 < ||Px|| + 5 [since clearly Opt < ||Px||] (23) 

(c) (f){Az) < (f){Ax -y) + (f){y-Ax)<e + 6^ + e = 2e + 6^ 
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We now have 



Thus, 



\Bx\\+5 > \\Bx\\ = \\Bx + Bz\\ = \\w + w + Bz\\ [by (23.fe)] 

> + — [by the triangle inequahty] 

> — ||w|| + \\w\\ + ||Pi?2;|| — ||Pi?2;|| [by (23. a) and Lemma 3.1] 

> -\\w\\ + \\w\\ + {l-j)\\Bz\\-mAz) [by C,(7,/3;0)]. 



(1-7)||5^|| < \\Bx\\+S -\\w\\ + \\w\\+ (3^{Az) <S + 2\\w\\+ /3[2e + S^] 

[by (23. c) and the triangle inequality; note that Bx = w + w] 
< 6 + 26^ + P[2e + 6^] [since \\w\\ = \\Bx - PBx\\ < 5J, 



and (9) follows. 



A. 2 Proof of Proposition 3.2 

Let X, P & Vs, X satisfy the premise of the proposition, and \etz = x — x,C,=y — Ax. We have 

\\Bx\\ + X(f){Ax -y) < \\Bx\\ + X(j){Ax - y) + S, 

(p{Ax-y) = ^{Az + Ax-y)>(P{Az)-(f){C), 

and we conclude that 

\\Bx\\ + \(f){Az) < \\Bx\\ + 2A0(O + S. (24) 

Further, 

\\Bx\\ = \\Bx + Bz\\ = \\PBx + {I - P)Bx + Bz\\ 

> \\PBx + Bz\\ — ||(/ — P)Bx\\ [by the triangle inequality] 

> -||(/ - P)Bx\\ + \\PBx\\ + \\PBz\\ - \\PBz\\ 

(the latter inequality is (5) as applied with w = PBx; note that Pw = w due to P^ = P). Thus, 
when applying condition 0^(7, </>), we obtain 

\\Bx\\ > -5^ + \\PBx\\ + {l--f)\\Bz\\- (3^{Az), 

and therefore 

-6^ + \\PBx\\ + {l--f)\\Bz\\- (3(j){Az) + X(j){Az) < \\Bx\\ + X(f){Az) 

< \\Bx\\+2X(j){^) + 6 

where the concluding < is due to (24). We come to 

(l-7)||fiz|| < \\Bx\\-\\PBx\\+5^ + [(3-X]<p{Az) + 2X(l)iO + 5 

<o 

< 26x + 2A0(^) + 6 [the triangle inequality] 
The resulting relation is nothing but (10). ■ 
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A. 3 Proof of Proposition 4.4 

Let us treat pq x pq matrices A as q^ p block matrices with p x g blocks A^^ , 1</U<q', 1<'^<P, 
so that ioi A = nF ®z we have A^" = w^^z. Now, let X = W^'^, and let L{X, X) be the family of all 
linear maps z — )■ Rz : X ^ X. A map R G L{X, X) can be identified with tensor Rijke, ^ ^ i, k < p, 
1 < < 95 according to [Rz]ij = ^ RykiZM- By definition of 0[-], for every R G X) and all 
z,w & X we should have 

Tt{Q[R][w'^ ® z]) = Tt{[Rz]w^) = RijMWijZki, 

i,j,k,£ 

SO that Q[R] is pq xpq matrix with the blocks (6[-R])'^'' = R^"^ := [R^^ke] i<k<p, G R^^'. In particular, 

l<£<g 

if 5 is a X pq matrix such that all blocks S'^" belong to the kernel of R, then Tr(6[i?]i5) = 
E,,.Tr(i?'^^[5n^) = 0. 

Now let S be the pq x pq matrix with the blocks S'^'^ possessing each exactly one nonzero entry, 
equal to 1, in the position ufi, so that S^'^ are just the standard basic orths in R^^"^. Denoting by 
yy/xi/^ y^iiu ^YiQ blocks in the pqxpq matrices G[iy] and G[i7*A], respectively, we have for all z,w & X: 

TT{£[w^(^zf) = Tt{zw^) = Tt{[{W + H*A)z]w^) = TT{e[W][w^ ® zf) + TT{e[H*A][w'^ ® zf), 

whence = W^"" + nf"" for all z/. 

Consider any Z E S where S is the set of all pq x pq matrices Z of nuclear norm < 1 and such 
that all blocks Z'^'^ belong to the kernel of the sensor map A. Then Z'^'^ belong to the kernel of H*A, 
so that by the above TiiW^lZ^'^f) = for all /i, whence Tt{£Z^) = Tt{Q[W]Z^). 

We clearly have 

0^[W] > Opt[W] := max {Tt{Q[W]Z^) : Z G RP'^'p'', \\Z\\ < 2s, \\Z\\2,2 < 1, G Ker(A) V^, z/} 
= max {Ti{£Z^) : Z G Rpi'^pi, \\Z\\ < 2s, \\Z\\2,2 <1,Z'"' e Ker(A) V/i, u} . 

Passing to the dual problem, we get 

Qpt[iy] = min {2s||W||2,2 + ||V|| : W, V G RPixp^^U"" + - £'"" G (Ker(A))^ V/i, u} . (25) 

Now let W, V be a feasible solution to the latter problem. Denoting by || ■ ||f the Frobenius norm on 
RPixpi, we have 

\ml<Pq\ml2, l|V||^< llVf. (26) 
On the other hand, (U + V^" - Sf"" G (Ker(A))^ V/i, u implies that 

+ VIII > ^dist^(^^^(Ker(yl))^), (27) 

where the distance is measured in the Frobenius norm. Let us treat R^^^ as a pg-dimensional 
Euclidean space R*^, M = pq, equipped with the standard Euclidean structure, and let d be the 
dimension of Ker(yl). Observe that the right hand side in (27) is sum, over all basic orths £^'^ of R*'^, 
of squared distances from the basic orths to a linear subspace L of R^^ of codimension d, namely, 
L = (Ker(74))"'". Note that such a sum is at least d. Indeed, otherwise we could approximate the 
unit M X M matrix by with a matrix of rank < M — d (specifically, a matrix with columns from L) 
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within an accuracy, measured in the Frobenius norm, which is better than v^, which is impossible. 
Thus, the right hand side in (27) is at least d = dim(Ker(yl)), and we arrive at the inequality 

||W + V||| > := dim(Ker(A)) ^ + ||V||f > v^. 

Invoking (26) and recalling that U,V is an arbitrary feasible solution to (25), we arrive at 

2s 



Opt[W] > Opt[W] > min 



a.,b>0 I ^pq 



+ b: a + b> v^dimKer^) | 



mm 



^^J dim(Ker(A)) ^ 
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